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ABSTRACT 

This study evaluated the effectiveness of an expert 
instructional system used to provide timely and quality feedback 
about the quality of a variety of visual materials production 
techniques to undergraduate students in a basic instructional 
materials production class. One group of students generated its own 
feedback by using the expert system, and the other group received 
feedback from a teacher. The expert system used a question and answer 
approach which required students to relate textual information to 
what they already knew, and to evaluate each production project based 
on a set of criteria for that proiect. The evaluation task was 
subjected to two measurements: (1) a content analysis, which was used 
to count the number* of standard criteria cited and the number of new 
criteria applied or generated; and (2) opinion measurements, which 
included a self-rating of learning and a rating of the expert system. 
Nether the quality measurement nor the citation of details 
measurement (except for evaluation of lettering samples) produced 
significantly higher results for the group using internally generated 
feedback. However, the internally generated feedback group produced a 
significantly greater number of non-standard criteria in the second 
measurement of the evaluation task. Overall, the opinion measurements 
of the students who used the expert system to generate their own 
feedback about their project were favorable. The text is supplemented 
by 2 figures, 5 tables, and 19 references. (EW) 
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Internally Generated Feedback 
with an Instructional Expert System 



Background 

The provision of feedback is a major part of instruction. This is especially 
important in a procedural task, such as the production of instructional ma- 
terials, where students learn and practice new skills with unknown expec- 
tations. Appropriate feedback confirms the learner's expectancy, directs at- 
tention to relevant factors, and stimulates recall of relevant skills and 
knowledge (Gagne, 1977). Generally, research indicates that "appropriate 
feedback" is precise, frequent, and most valuable immediately following 
completion of a task (Van Houten, 1980). However, the feedback process is 
often highly dependent upon the quality of the teacher administering the 
instructional system. 

The impetus for this study arose from the need to provide timely and spe- 
cific feedback about the quality of a variety of visual materials production 
techniques (dry mounting, laminating, rubber cement mounting, lettering, 
and cutting). The provision of appropriate feedback was a persistent prob- 
lem because minimally qualified teaching assistants (TAs) were used to staff 
the class lab sessions. The TAs main weaknesses were in the areas of 
consistency, accuracy, and knowledge of production techniques. Students 
frequently had to wait long periods for advice on the quality of their skills 
and projects. There was also a long gap between completion of a project 
and its evaluation report. While the learning of procedural skills was for 
the most part successful, there were wide differences in quality and the 
ability of the students to evaluate their own or others application of the 
skills... 

The purpose, then, of this study was to develop an instructional system 
that would provide precise and timely feedback. More specifically, the ob- 
jectives of this project included both basic and applied concerns: first, was 
the need for a teaching strategy that would facilitate the learning of pro- 
duction skills and their subsequent transfer to an evaluation and self -cri- 
tiquing task; and second, was the need to develop an instructional system 
that would efficiently and effectively teach a large number of undergradu- 
ates with minimally qualified teaching assistants. 

The proposed solution called for an expert system as a central part of a 
complete instructional system. An expert system seemed a creative and 
effective because of the likelihood of a change in student processing strate- 
gies. It was believed that the expert system would enable the students to 
use processing strategies not used when receiving feedback from TAs. The 
TAs, who facilitated learning by offering suggestions and brief critiques of 
student techniques, did not foster the growth of a schemata appropriate for 
learning to evaluate productions because their method did not illuminate 
nor involve the student in using the evaluation criteria. The expert sys- 
tem, on the other hand, offered a method that would illuminate the pro- 
duction criteria by taking learners through questions, choices and decisions 
in a step-by-step manner. It was expected that the expert system would 
permit the active manipulation of variables (production criteria) while 
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testing the affects of hypotheses (production practices) would affect learning 
(Salomon, 1985). Salomon (1985) stated that 

"The more a technique can be designed to activate, model, or 
short-circuit cognitions hitherto untapped by other means, 
the greater its opportunity for fostering unique learning ef- 
fects and for making a difference in learning and develop- 
ment." (p. 210) 

In this case, the expert system embedded strategies in the instructional pre- 
sentation that would encourage the learner to process the information in a 
systematic and precise way in order to work through the material (Rigney, 
1978). In other words, the expert system fostered a more "mindful" 
(Salomon, 1985) approach to instruction on the part of the learner by di- 
recting attention and encouraging a more mentally active role in evaluating 
their own projects. The expert system also moved the focus of feedback 
provision from external to internal. The students, in a sense, generated 
their own feedback rather than having it provided externally from the 
teaching assistant. Hillman (1970) showed that a group receiving immediate 
feedback through self-scoring showed considerably more improvement oh a 
standard test than the group receiving 24-hour feedback. 

The proposal to use an expert system under such conditions raised several 
questions. First, could an expert system provide feedback as effectively as 
a "skilled" instructor? Second, how would the use of an expert system af- 
fect the learning of procedural objectives? Third, how would the use of an 
expert system affect the learning of the ability to apply the production 
skills to an evaluation task? Finally, what would be the reactions and 
opinions of students using a "machine" for feedback? The following hy- 
potheses were formed to guide the study: 

1. The learning and application of procedural production skills would be 
equal between a group using the expert system for feedback and a 
group receiving teacher provided feedback. No difference in procedural 
skills was expected, despite the feedback system because both groups 
would be instructed by the same teacher in the same manner and 
complete the same assignments. Clark (1984, 1985) and Hagler and 
Knowlton (1987) emphasized that if the message is the same the learn- 
ing will be the same. 

2. A group using an expert system for feedback would provide more de- 
tailed evaluations of new projects than a group receiving teacher feed- 
back. The contention is that students using an expert system would 
receive feedback, but would utilize different processing strategies. If 
different processing strategies are used, then it is likely that different 
learning will result. It seemed likely that the expert system users 

. .would learn more, of the details about the criteria used to evaluate the 
projects since they would be forced to work through the criterial in 
detail and this difference would show up on an evaluation task. 

3. Expert system users, would have a positive rttitude about using the 
system. Several studies (Hess and Tenezakis, 1973; Schurdak, 1967; and 
Schwartz and Long, 1967) have found that CAI is frequently preferred 
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by learners, therefore it seemed reasonable to expect that the students 
would have positive opinions regarding the use of the expert system. 



Method 

Subjects; 

Subjects were 43 undergraduate students from a basic instructional mate- 
rials production class that was a requirement in the teacher preparation 
program. Ages ranged from 20 to 40 years of age. There were 25 women 
and 18 men. 

Treatment Groups: 

Two treatment groups were used: one group generated itgs own feedback 
by using an expert system and the other group received feedback from a 
teacher. 

Procedures; 

The subjects, though from one class, belonged to two different labs: a 
morning lab and an afternoon lab. To minimize confounding (Kulik, Kulik, 
and Cohen, 1980; Clark, 1985; and Hagler and Knowlton, 1987) both groups re- 
ceived the same instruction at the same time from the same teacher. 

After attending the morning lecture, the first lab group went directly to a 
lab session while the afternoon lab left and returned at their appointed 
time. The treatments were assigned to the lab sections: the morning lab 
received the expert system feedback treatment and the afternoon lab re- 
ceived the human feedback treatment. Each student performed five pro- 
duction tasks: dry mounting, laminating, rubber cement mounting, letter- 
ing, and cutting. 

Labs were used as practice and production time. For both groups, the only 
assistance given by the instructor was in cases of correcting improper or 
unsafe techniques. When the expert system group had questions regarding 
the quality of a particular technique or project, they were required to use 
the expert system for advise. When the teacher-provided feedback group 
had questions regarding the quality of their work they were answered by 
the teacher. 

One week following completion of the production tasks they evaluated a 
new sample of each production task. The evaluation task took the form of 
a written essay whereupon the student wrote as much as they chose about 
. the strengths and weaknesses of each of five project samples. 

Expert System Materials: 

The expert system was created with an a shell, Expert Ease (Human Per- 
formance Technolgy, 1984) and used a question and answer approach. Stu- 
dents had to read the questions, analyze their work, make an evaluative 
Judgment, and enter responses to work through the program. The ques- 
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tion/response technique was used because questions that require learners to 
relate textual information to what they already know, to interpret and in- 
fer, make the textual material more meaningful and more memorable 
(Lindner and Rickards, 1985). There was a separate section within the ex- 
pert system for each production project. Each section was composed of a 
series of questions based on the criteria (see Figure 1 for a summary of the 
criteria for each project) used to evaluate each project (see Figure 2 for two 
screen samples from the expert system program). At the end of each crite- 
rion the student was asked to Judge whether the project was satisfactory 
or unsatisfactory with respect to that criterion. 

Testing and MeasuronignU; 

Each of the five production projects were evaluated on a five point scale. 
Evaluators used a checklist comprised of the standard criteria for each 
project. The evaluation task was subjected to two measurements. Within 
each of the five written project evaluations content analysis was used to 
count the number of standard criteria cited and of the number of new cri- 
teria applied or generated. Opinion measurements included a self-rating of 
learning, a rating of the expert system. 



Results 

Table 1 reports the results of the quality of the production projects when 
compared using a *wo-tailed t-test (Chase, 1976). Each project was evalu- 
ated on a 10 point scale (10 was excellent). Although each preset mean for 
the internally generated feedback group was higher than the corresponding 
project mean for the externally provided feedback group, the differences 
were not significant (p < .05). 

Table 2 reports the t-test results for the number of details that the partici- 
pants cited from the standard criteria in their written evaluations of sam- 
ple projects in the evaluation task. The only significant difference (p < .05) 
in the ability of the two groups to evaluate a project was in the evaluation 
of lettering samples. The internally generated feedback group (2.9333) cited 
significantly more details than the group receiving external feedback 
(2.5455). 

The second measure of the evaluation task, the number of non-standard 
criteria generated, showed a significant effect (p < .05) for the internally 
generated feedback group (see Table 3). The internal feedback group gener- 
ated an average of 1.6 new items for all five evaluation tasks, while the 
externally provided feedback group provided generated an average of 1.3 
new details. 

Overall, the opinion measurements of the students who used the expert 
system to generate their own feedback about their projects were favorable. 
Table 4 shows that_the students believed that the knowledge they gained 
was essentially the same as the externally provided feedback group. They 
also believed that the use of the expert system neither improved nor hurt 
their project evaluation scores. Table 5 presents the summary of results 
for overall reaction to using the expert system to generate feedback about 
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their projects. Two-thirds of the reactions were at level six or above on 
the 10-point scale (10 was positive). 



Discussion 

As predicted, both instructional methods were equally effective in the 
teaching of the procedural skills used for producing the projects. This re- 
sult is consistent with Clark's (1984) statements that learning comes from 
effective instructional design — not from the medium used to deliver the 
instruction. Note, though, in Table 1, all of the project evaluation means of 
the internally generated feedback group are higher than the externally pro- 
vided feedback group. Is this trend due to the novelty effect? Probably not 
because the use of computers for instruction was not new to the class. Did 
the students who used the expert system try harder? Again, probably not, 
because this was the first unit in the first week of a new session — a time 
when most students are motivated and ready to begin anew. The trend 
may be due to the positive and specific nature of the questions asked by the 
expert system. The expert system provided information related to both 
successes and failures, while the lab assistant usually concentrated on fail- 
ures of production technique. The expert system provided a means to gen- 
erated feedback that was non-threatening as well as specific and positive. 
The feedback generated through the expert system was also timely, stu- 
dents used the expert system either during or after production at a time 
when corrective feedback is most valuable (Van Houten, 1980). 

There were also two indications that internally generated feedback encour- 
aged a different kind of processing and learning. The first indication of this 
was the significant difference between the two groups on their evaluations 
of the lettering projects. This may be due to the complexity of the lettering 
project. The lettering project had seven rather unfamiliar criteria to 
learn, while each of the four other projects had only five similar criteria 
(see Figure 1). The step-by-step method of analyzing each criterion with 
the expert system may have given the participants more memorable prac- 
tice. The second indication was the ability of the internally generated feed- 
back group to be more creative in their evaluations as represented by their 
ability to cite more than the standard criteria and to apply criteria from 
one project to another. This seems to indicate a more thorough learning of 
all the criteria and their applications. 

Finally, the opinions of the students regarding the use of the expert system 
as part of the instructional program was positive. The expert system was 
readily available, it was a non-threatening objective evaluation procedure, 
and permitted the students to take independent responsibility for a major 
part of their learning process. 

It is not the contention here that the reason for the results is the medium 
— only that an expert system can be an effective part of a well designed 
instructional system, "the results could probably be duplicated with more 
thorough TA training or student checklists or TA Job aids. The computer 
provides the unique opportunity to demand a response (Avner, Moore, and 
Smith, 1980) before moving on — it is a way to prevent short cuts through 
the designed learning processes. However, the expert system provided an 
instructional method that placed the responsibility for learning on the sta- 
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dent. The internally generated feedback proved as effective as the exter- 
nally provided feedback for the procedural tasks and better than the exter- 
nal feedback in a declarative task such as evaluation. In ether words the 
medium was more suited to the objectives for the learner (Allen and Mer- 
rill, 1985; Hagler and Knowlton, 1987) in this situation. 

The reliance on internally generated feedback with the expert system raises 
several additional questions. A basic question is whether the expert system 
that copies or models an expert's reasoning actually teaches the same rea- 
soning process to the users. It would also be interesting to examine the im- 
plementation of a more complex expert system, a system that is composed 
of many questions or that attempts to actually grade a project on a wider 
scale than pass/fail. The task complexity is also another factor worth in- 
vestigation. Writing, drawing, or problem solving tasks may also have 
roles for internally generated feedback. 
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Figure 1 



Production Criteria for Projects 
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letter spacing 
word spacing 
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layout 
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stray adhesive stray adhesive stray adhesive 
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Figure 2 
Expert System Shell 3a nplt 



Neatness Criterion for the Easel Project 

OVERALL NEATNESS: 

Neatness is a high priority for all projects. 
It 2s a prerequisite, meaning that this criterion 
must be met before any others will be considered. 

PENCIL MARKS: Wh?n marking the boundaries of your easel 

use as few pencil marks as possible. And, make 
the marks as light as possible for easy erasure. 
There should be NO visib>? pencil marks on the 
easel. 



ERASURES: if you need to eraso pencil marks make sure that 

there are no erasures left on the easel. 

MISCELLANEOUS MARKS: Avoid grease, coffee stains, and any other 

material that may make your easel look sloppy. 

1) "SATISFACTORY" means that you have a neat, clean and professional 

appearing product. 

2) "UNSATISFACTORY" means that you have one or more of the 
p roblems described above. 

Sturdlness Criterion for Easel Project 

STURDINESS: 



The aasel should be sturdy and solid. If the easel 
bends or "bows" as it sits up it is not sturdy enough. 
Generally, you should not use poster board, which is 
a bit thin for easels, especially those taller than 
three inches. You can solve this problem by dry mounting 
two pieces of poster board together to make a thicker and 
stronger piece of material from which to cut the easel, 
or by using corrugated cardboard. 

1) "SATISFACTORY" means that you have a neat, clean and professional 

appearing product. 

2) "UNSATISFACTORY" metns that you have.one or more of the 

problems described above. 
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Table 1 
Project Scores 





Expert System 


Teacher 


Sijmlf. 


Lamination 


7.9048 


7.3478 


.603 


Rubber Cement 


7.3333 


6.9130 


.700 


Dry Mount 


7.9524 


7.6087 


.187 


Cut 


7.1905 


6.5652 


.146 


Lettering 


7.1905 


6.7391 


.372 


Overall Score 


7.5143 


7.0348 


.672 



Table 2 

Number of Details Cited in Evaluation Task 





Expert System 


Teacher 


Sianif. 


Lamination 


3.5333 


3.0455 


.618 


Rubber Cement 


2.8000 


2.1818 


.779 


Dry Mount 


2.4667 


2.5909 


.740 


Cut 


2.5333 


2.5909 


.301 


Lettering 


2.9333 


2.5455 


.028* 



Table 3 

Extra Details Mentioned on Written Evaluation Task 

, Expert System Teacher Si «nif. 

Total No. Extra 1.6000 1.3043 .015* 
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Table 4 
Descriptive Summaries 

More Same Less 

Amount of Learning 5 16 0 

Better Same Bad 

Affect on Project Score 4 15 2 



Table 5 

Overall Reaction to Expert System 

negative 0123456789 10 pos. 
Reaction . 1 2 3 4 5 4 2 

Percent 5 9.5 14 19 24 19 9.5 
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